Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Telemetry] Data: Report dataset info only if there is known metadata #71419

Merged
merged 5 commits into from
Aug 3, 2020
Merged

[Telemetry] Data: Report dataset info only if there is known metadata #71419

merged 5 commits into from
Aug 3, 2020

Conversation

afharo
Copy link
Member

@afharo afharo commented Jul 13, 2020

Summary

On top of #64935, this PR adds the collection of the constant_keywords dataset.name and dataset.type only if Elastic-known meta fields are found in the index's mappings.

We'll collect the information about the dataset fields mentioned above only if _meta.beat (reported as shipper) exists, or the index is maintained by the new Ingest Manager (it should contain _meta.managed_by === 'ingest-manager' and/or the _meta.package.name` field).

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@afharo afharo marked this pull request as ready for review July 13, 2020 13:06
@afharo afharo requested a review from a team as a code owner July 13, 2020 13:06
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-telemetry (Team:KibanaTelemetry)

@afharo afharo added the release_note:skip Skip the PR/issue when compiling release notes label Jul 13, 2020
Bamieh
Bamieh approved these changes Jul 13, 2020
View reviewed changes
Copy link
Member

@Bamieh Bamieh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@TinaHeiligers
Copy link
Contributor

@afharo it's FF today. Please merge if possible, otherwise we'll have to bump to 7.10.

@afharo
Copy link
Member Author

afharo commented Jul 15, 2020

Update: This PR is currently blocked until elastic/elasticsearch#59195 is done.

There is no point on checking the metadata to retrieve the dataset.* info if data-streams won't have _meta information just yet.

@afharo afharo mentioned this pull request Jul 15, 2020
Closed
@afharo afharo added v7.10.0 and removed v7.9.0 labels Jul 15, 2020
@afharo
Copy link
Member Author

afharo commented Jul 15, 2020

Replanning for 7.10 as tentative (but it still depends on the ETA of elastic/elasticsearch#59195)

@afharo afharo mentioned this pull request Jul 16, 2020
Merged
2 tasks
@afharo
Copy link
Member Author

afharo commented Jul 20, 2020

@elasticmachine merge upstream

@afharo
Copy link
Member Author

afharo commented Jul 22, 2020

@TinaHeiligers and @Bamieh! After testing the new Ingest Manager, I noticed we had to add an exception for the way the data-streams actually work and seen by Kibana:

  • Data Stream name: logs-nginx.access-default
  • Actual index name: .ds-logs-nginx.access-default-000001

Funny enough: the data-stream works as an alias, so you can get the index .ds-logs-nginx.access-default-000001 with the request GET {DATA_STREAM_NAME} or, obviously, GET {INDEX_NAME}.

That's why in my last commit I had to change it to GET *-*-* (instead of GET *-*-*-*) and allow indices starting with .ds- when filtering out the system indices.

I'll wait for you to confirm if you are happy with these latest additions :)
Thank you!

@afharo
Copy link
Member Author

afharo commented Aug 3, 2020

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@afharo afharo merged commit df41c9b into elastic:master Aug 3, 2020
@afharo afharo deleted the telemetry/data/dataset-if-known-meta branch August 3, 2020 17:07
@afharo afharo mentioned this pull request Aug 3, 2020
Merged
afharo added a commit that referenced this pull request Aug 3, 2020
@afharo @elasticmachine
[7.x] [Telemetry] Data: Report dataset info only if there is known me…
78a4824 
…tadata (#71419) (#74126)

* [Telemetry] Data: Report dataset information only if there is known metadata

* Handle data-stream indices (.ds-*)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@lukeelmers lukeelmers added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Oct 1, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Bamieh Bamieh Bamieh approved these changes

Assignees
No one assigned
Labels
Feature:Telemetry release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.10.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@afharo @elasticmachine @TinaHeiligers @kibanamachine @Bamieh @lukeelmers